Audio transitions when streaming audiovisual media titles

ABSTRACT

A playback application is configured to analyze audio frames associated with transitions between segments within a media title to identify one or more portions of extraneous audio. The playback application is configured to analyze the one or more portions of extraneous audio and then determine which of the one or more corresponding audio frames should be dropped. In doing so, the playback application can analyze a topology associated with the media title to determine whether any specific portions of extraneous audio are to be played outside of a logical ordering of audio samples set forth in the topology. These specific portions of extraneous audio are preferentially removed.

BACKGROUND Field of the Various Embodiments

Various embodiments relate generally to computer science and, morespecifically, to improved audio transitions when streaming audiovisualmedia titles.

Description of the Related Art

A video streaming service is typically designed to provide users withaccess to one or more libraries of various media titles. To access agiven media title, a user usually connects to the video streamingservice via an endpoint device, such as a laptop computer, smarttelevision, tablet computer, or similar device. The user can then selectthe given media title via a graphical user interface (GUI) that isdisplayed on the endpoint device and configured to allow users to makeselections from a particular library of media titles. Upon selecting thegiven media title, the video streaming service streams media contentassociated with the media title to the endpoint device. The streamedmedia content normally includes encoded frames of video and/or audiothat can be decoded by a media player executing on the endpoint deviceto provide video and/or audio output to the user.

Before a given media title is available for streaming, the videostreaming service usually performs encoding operations on source mediacontent to generate one or more different encoded versions of the sourcemedia content. For example, the video streaming service could encode thesource media content with a range of different video and/or audioresolutions. Encoding the source media content in this fashion allowsthe video streaming service to stream different versions of the mediatitle to users based on differing levels of available bandwidth and/orother variable streaming parameters.

The source media content typically includes raw video frames and rawaudio samples corresponding to those raw video frames. Differentsequences of raw video frames and corresponding raw audio samples aresometimes organized into different “segments.” A given segment could be,for example, an introductory portion of a serial media title, a portionof a story arc associated with a branching narrative media title, aspecific scene within a given media title, a short segment of paddingthat is included for various reasons but is not meant to be rendered, ora credit sequence associated with a given media title, among others.Multiple segments are oftentimes stored together in memory in a “back toback” arrangement without any intervening gaps between adjacent rawvideo frames or between adjacent raw audio samples. Physical boundariesbetween segments are usually defined based on specific raw video framesthat delineate the start and end of any two physically adjacentsegments.

When encoding raw video frames, a video encoder sequentially encodes theraw video frames included in a given segment to generate encoded videoframes for that segment. The video encoder typically generates adifferent encoded video frame for each raw video frame in the segment,including the specific raw video frame that delineates the end of thesegment. When encoding raw audio samples, an audio encoder sequentiallyencodes groups of raw audio samples included in the given segment togenerate encoded audio frames for that segment. The audio encodertypically generates a different encoded audio frame for each group ofraw audio samples in the segment.

When performing the above encoding operations with certain encodersettings, the number of raw audio samples included in a given segmentmay not be evenly divisible by the number of raw audio samples includedin a given group of raw audio samples. In such cases, when the audioencoder reaches the end of the given segment, not enough raw audiosamples remain to form a complete group. Consequently, the audio encoderusually groups the remaining raw audio samples from the end of the givensegment with some of the raw audio samples from the beginning of thesegment of the source media content that resides physically subsequentto the given segment in memory. The audio frame generated using thesedifferent audio samples effectively overlaps the physical boundarybetween the given segment and the physically-subsequent segment inmemory. During playback of the given segment, the overlapping audioframe is decoded in conjunction with playback of the last video frame ofthe given segment. Similarly, during playback of thephysically-subsequent segment, the overlapping audio frame is decoded inconjunction with playback of the first video frame of thephysically-subsequent segment. In both cases, the overlapping audioframe is decoded to generate some audio samples associated with the lastvideo frame of the given segment and other audio samples associated withthe first video frame of the physically-subsequent segment.

When the given segment and the physically-subsequent segment are playedconsecutively to one another, the overlapping audio frame is decoded inconjunction with a transition between the last video frame of the givensegment and the first video frame of the physically-subsequent segment.In this situation, audio playback occurs “normally” across thetransition between segments because the overlapping audio frame isdecoded to produce some audio samples associated with the video datapresent at the end of the given segment and other audio samplesassociated with video data present at the beginning of thephysically-subsequent segment. However, when the given segment and asegment of the source media content that does not reside physicallysubsequent to the given segment in memory are played consecutively toone another, the overlapping audio frame is decoded in conjunction witha transition between the last video frame of the given segment and thefirst video frame of the non-physically-subsequent segment. In thissituation, audio playback does not occur “normally” across thetransition between segments because the overlapping audio frame isdecoded to produce some audio samples associated with the video datapresent at the end of the given segment and other audio samples that areassociated with the video data present at the beginning of thephysically-subsequent segment instead of with the video data present atthe beginning of the non-physically-subsequent segment.

One drawback of the above approach is that users may notice aninconsistency between the video that is being played back and the audiothat is being played back. These types of inconsistencies between thevideo being played back and the audio being played back can be confusingto users and potentially result in a poor user experience. Anotherdrawback is that the correct audio samples that are played back may beacoustically inconsistent with the incorrect audio samples that are tobe played subsequent to the correct audio samples. For example, thecorrect audio samples could have a low amplitude and, therefore, wouldbe played back with a lower volume; whereas, the incorrect audio samplescould have a high amplitude and, therefore, would be played with ahigher volume. These types of inconsistencies in audio that is playedback can be jarring to users and also potentially result in a poor userexperience.

As the foregoing illustrates, what is needed in the art are moreeffective techniques for transitioning across different segments of amedia title.

SUMMARY

Various embodiments include computer-implemented method, includingidentifying a first portion of audio data that should not be played backin conjunction with a first sequence of video segments associated with amedia title, identifying a second portion of audio that also should notbe played back in conjunction with the first sequence of video segments,determining that the first portion of audio data has a greater playbackduration than the second portion of audio data, and preventing the firstportion of audio data from being played back in conjunction with thefirst sequence of video segments.

One advantage of the disclosed techniques is that portions of audio datathat are not meant to be played back in conjunction with video framesderived from one or more segments of a media title can be moreeffectively reduced. Accordingly, situations where incorrect audio isoutput during playback of a given segment can be largely avoided,thereby enhancing the overall quality of the user experience whenstreaming media titles.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the variousembodiments can be understood in detail, a more particular descriptionof the inventive concepts, briefly summarized above, may be had byreference to various embodiments, some of which are illustrated in theappended drawings. It is to be noted, however, that the appendeddrawings illustrate only typical embodiments of the inventive conceptsand are therefore not to be considered limiting of scope in any way, andthat there are other equally effective embodiments.

FIG. 1 illustrates a network infrastructure configured to implement oneor more aspects of the various embodiments;

FIG. 2 is a block diagram of a content server that may be implemented inconjunction with the network infrastructure of FIG. 1, according tovarious embodiments;

FIG. 3 is a block diagram of a control server that may be implemented inconjunction with the network infrastructure of FIG. 1, according tovarious embodiments;

FIG. 4 is a block diagram of an endpoint device that may be implementedin conjunction with the network infrastructure of FIG. 1, according tovarious embodiments;

FIG. 5A illustrates how the content of FIG. 4 is organized in memory,according to various embodiments;

FIG. 5B illustrates how the topology of FIG. 4 defines different logicalsequences of segments, according to various embodiments;

FIG. 6A illustrates a logical sequence of segments that is consistentwith a physical sequence of those segments in memory, according tovarious embodiments;

FIG. 6B illustrates a logical sequence of segments that is inconsistentwith a physical sequence of those segments in memory, according tovarious embodiments;

FIG. 7A illustrates how the playback application of FIG. 4 selectivelydrops audio frames to reduce the playback of extraneous audio data,according to various embodiments;

FIG. 7B illustrates how the playback application of FIG. 4 drops andcrossfades audio frames to minimize the playback of extraneous audiodata, according to various embodiments;

FIG. 7C illustrates how the playback application of FIG. 4 drops andshifts audio frames to minimize the playback of extraneous audio data,according to various embodiments;

FIG. 8 is a flow diagram of method steps for reducing the playback ofextraneous audio data, according to various embodiments;

FIG. 9 illustrates how the playback application of FIG. 4 reducesextraneous audio data based on a media title topology, according tovarious embodiments;

FIG. 10A illustrates how the playback application of FIG. 4 duplicatesan audio frame to avoid playing back extraneous audio data, according tovarious embodiments;

FIG. 10B illustrates how the playback application of FIG. 4 moves atransition between audio frames to avoid playing back extraneous audiodata, according to various embodiments;

FIG. 11A illustrates how the content of FIG. 4 can be organized inmemory, according to various other embodiments;

FIG. 11B is an alternative illustration of the topology of FIG. 4,according to various other embodiments;

FIG. 12A illustrates a logical sequence of segments within the topologyof FIG. 11B, according to various embodiments;

FIG. 12B illustrates another logical sequence of segments within thetopology of FIG. 11B, according to various embodiments;

FIG. 13 is a flow diagram of method steps for reducing the playback ofextraneous audio data, according to various other embodiments;

FIG. 14A illustrates raw media content that is encoded to generate thecontent of FIG. 4, according to various embodiments;

FIG. 14B illustrates various sequences of segments associated with thetopology of FIG. 4, according to various embodiments; and

FIG. 15 is a flow diagram of method steps for modifying raw audiosamples to reduce the playback of extraneous audio data, according tovarious embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the various embodiments.However, it will be apparent to one of skill in the art that theinventive concepts may be practiced without one or more of thesespecific details.

A video streaming service provides users with access to various types ofmedia titles. A user can stream media content associated with a selectedmedia title to an endpoint device. The media content includes encodedframes of video and/or audio. A media player executing on the endpointdevice performs various decoding operations with the media content andthen outputs decoded video and/or audio to the user. The quality of thedecoded video and/or audio depends, at least to some degree, on theencoding procedure used to generate the encoded frames of video and/oraudio included in the media content. Delivering high-quality mediacontent is an important cornerstone in the ongoing effort to maintainand expand the current user base. However, various technical issuesassociated with conventional encoding procedures impede the delivery ofhigh-quality media content.

In particular, a conventional encoding procedure is applied to sourcemedia content to generate encoded frames of video and/or audio that canbe streamed to users. The source media content is organized intodifferent viewable “segments” that include sequences of raw video framesand corresponding raw audio samples. During a video encoding portion ofthe conventional encoding procedure, a video encoder processes the rawvideo frames to generate video frames encoded with a particular videoencoding standard. During an audio encoding portion of the conventionalencoding procedure, an audio encoder processes groups of raw audiosamples according to a particular audio encoding standard to generateaudio frames corresponding to those video frames.

With certain encoder settings, the audio encoder can group together rawaudio samples that reside on either side of a boundary between segmentsand generate an audio frame that overlaps or straddles the boundarybetween segments. Some of the raw audio samples correspond to the lastvideo frame of a first segment, while other raw audio samples correspondto the first frame of a second segment that resides physicallysubsequent to the first segment in memory.

When the first segment and the second segment are played successively,the overlapping audio frame is decoded in conjunction with a transitionbetween the last video frame of the first segment and the first videoframe of the second segment to produce a first set of audio samples. Thefirst set of audio samples is associated with both the first and secondsegments.

When the first segment and a third segment are played successively,though, that same overlapping video frame is decoded in conjunction witha transition between the last video frame of the first segment and thefirst video frame of the third segment to produce the first set of audiosamples mentioned above. A subset of those audio samples are associatedwith the second segment and therefore should not be played inconjunction with the third segment. Audio samples that can be played inconjunction with incorrect video frames are referred to herein as“extraneous audio.”

One drawback of the above approach is that extraneous audio is usuallyinconsistent with the video frames being played. Inconsistencies betweenaudio and video can be noticeable to users and can result in a poor userexperience. Another drawback of the above approach is that extraneousaudio can be acoustically inconsistent with the audio that should beplayed in conjunction with the transition between segments. Acousticallyinconsistent audio can be jarring to users and can lead to a poor userexperience.

To address these issues, various embodiments include a playbackapplication that is configured to analyze the timing of audio framesassociated with transitions between segments within a media title toidentify one or more portions of extraneous audio. The one or moreportions of extraneous audio correspond to one or more audio frames thatcan potentially be decoded to play incorrect audio samples inconjunction with one or more frames of video. A given audio sample maybe considered “incorrect” when the given audio sample is not meant to beplayed back in conjunction with a particular frame or frames of videoand is instead supposed to be played back in conjunction with adifferent frame or frames of video. In other words, the given audiosample may be considered “incorrect” when the given audio sample isplayed back at the wrong time. The playback application is configured toanalyze the timing of one or more portions of extraneous audio and thendetermine which of the one or more corresponding audio frames should bedropped. Depending on endpoint device capabilities and/orconfigurations, the playback application can then perform varioustechniques to play back the remaining audio portions of differentsegments in a manner that reduces extraneous audio and/or audioartifacts.

The playback application is further configured to analyze a topologyassociated with the media title to determine whether any specificportions of extraneous audio are to be played outside of a logicalordering of audio samples set forth in the topology. The topology setsforth a directed graph of possible transitions between segments andconstrains playback of those segments to occur along those possibletransitions. The playback application determines any portions ofextraneous audio that conflict with the topology and then performsvarious techniques for eliminating these portions of extraneous audio.Those techniques include dropping one or more audio frames, duplicatingone or more audio frames across the transition between segments, and/ormoving an audio transition.

In addition to the foregoing techniques, various embodiments include apreprocessing engine that modifies the raw audio samples associated withthe media title prior to encoding in order to avoid the occurrence ofextraneous audio. The preprocessing engine analyzes the topology of themedia title in conjunction with analyzing the physical layout of themedia title in memory. Based on these analyses, the preprocessing engineidentifies pairs of segments of the media title that are logicallysequenced according to the topology but not physically sequenced inmemory. Extraneous audio can potentially be played during a transitionbetween any given pair of segments that can be identified in thismanner. The preprocessing engine analyzes a set of raw audio samplesassociated with one of the segments associated with a given pair ofsegments and then replaces a subset of those raw audio samples withanother subset of raw audio samples associated with the other segment inthe pair. The set of raw audio samples can subsequently be encoded toproduce an audio frame that does not decode to produce audio samplesthat should not be played in conjunction with a transition between thetwo segments. In this manner, the preprocessing engine can preemptivelyavoid playback of extraneous audio.

At least one technological advantage of the disclosed techniquesrelative to the prior art is that portions of audio data that are notmeant to be played back in conjunction with video frames derived fromone or more segments of a media title can be more effectively reduced.Accordingly, situations where incorrect audio is output during playbackof a given segment can be largely avoided, thereby enhancing the overallquality of the user experience when streaming media titles. Anothertechnological advantage of the disclosed techniques relative to theprior art is that portions of audio data that are acoustically divergentfrom other portions of audio data can be more effectively reduced whenanalyzed relative to the logical topology of a given media title.Accordingly, situations where jarring and unexpected audio data areinadvertently output to the user can be largely avoided, therebyenhancing the overall streaming experience. Yet another technologicaladvantage of the disclosed techniques relative to the prior art is thatextraneous audio data can be preemptively removed from raw audio samplesand replaced with non-extraneous audio. Accordingly, the occurrence ofincorrect audio during playback of a given media title can be reduced,further improving the overall quality with which media titles can bestreamed. These technological advantages represent one or moretechnological advancements relative to prior art approaches.

System Overview

FIG. 1 illustrates a network infrastructure configured to implement oneor more aspects of the various embodiments. As shown, networkinfrastructure 100 includes content servers 110, control server 120, andendpoint devices 115, each of which are connected via a communicationsnetwork 105. Network infrastructure 100 is generally used to distributecontent to content servers 110 and endpoint devices 115.

Each endpoint device 115 communicates with one or more content servers110 (also referred to as “caches” or “nodes”) via network 105 todownload content, such as textual data, graphical data, audio data,video data, and other types of data. The downloadable content, alsoreferred to herein as a “file,” is then presented to a user of one ormore endpoint devices 115. In various embodiments, endpoint devices 115may include computer systems, set top boxes, mobile computer,smartphones, tablets, console and handheld video game systems, digitalvideo recorders (DVRs), DVD players, connected digital TVs, dedicatedmedia streaming devices, (e.g., the Roku® set-top box), and/or any othertechnically feasible computing platform that has network connectivityand is capable of presenting content, such as text, images, video,and/or audio content, to a user.

Each content server 110 may include a web-server, database, and serverapplication 217 configured to communicate with control server 120 todetermine the location and availability of various files that aretracked and managed by control server 120. Each content server 110 mayfurther communicate with cloud services 130 and one or more othercontent servers 110 in order “fill” each content server 110 with copiesof various files. In addition, content servers 110 may respond torequests for files received from endpoint devices 115. The files maythen be distributed from content server 110 or via a broader contentdistribution network. In some embodiments, content servers 110 enableusers to authenticate (e.g., using a username and password) in order toaccess files stored on content servers 110. Although only a singlecontrol server 120 is shown in FIG. 1, in various embodiments multiplecontrol servers 120 may be implemented to track and manage files.

In various embodiments, cloud services 130 may include an online storageservice (e.g., Amazon® Simple Storage Service, Google® Cloud Storage,etc.) in which a catalog of files, including thousands or millions offiles, is stored and accessed in order to fill content servers 110.Cloud services 130 also may provide compute or other processingservices. In one embodiment, cloud services 130 may include apreprocessing engine 132. Preprocessing engine 132 is configured tomodify raw audio samples associated with media titles in order to reducethe playback of extraneous audio, as described in greater detail belowin conjunction with FIGS. 14A-15. Although only a single cloud services130 is shown in FIG. 1, in various embodiments multiple cloud services130 may be implemented.

FIG. 2 is a block diagram of a content server that may be implemented inconjunction with the network infrastructure of FIG. 1, according tovarious embodiments. As shown, content server 110 includes, withoutlimitation, a central processing unit (CPU) 204, a system disk 206, aninput/output (I/O) devices interface 208, a network interface 210, aninterconnect 212, and a system memory 214.

CPU 204 is configured to retrieve and execute programming instructions,such as server application 217, stored in system memory 214. Similarly,CPU 204 is configured to store application data (e.g., softwarelibraries) and retrieve application data from the system memory 214.Interconnect 212 is configured to facilitate transmission of data, suchas programming instructions and application data, between CPU 204,system disk 206, I/O devices interface 208, network interface 210, andsystem memory 214. I/O devices interface 208 is configured to receiveinput data from I/O devices 216 and transmit the input data to CPU 204via interconnect 212. For example, I/O devices 216 may include one ormore buttons, a keyboard, a mouse, and/or other input devices. I/Odevices interface 208 is further configured to receive output data fromCPU 204 via interconnect 212 and transmit the output data to I/O devices216.

System disk 206 may include one or more hard disk drives, solid statestorage devices, or similar storage devices. System disk 206 isconfigured to store non-volatile data such as files 218 (e.g., audiofiles, video files, subtitles, application files, software libraries,etc.). Files 218 can then be retrieved by one or more endpoint devices115 via network 105. In some embodiments, network interface 210 isconfigured to operate in compliance with the Ethernet standard.

System memory 214 includes a server application 217 configured toservice requests for files 218 received from endpoint device 115 andother content servers 110. When server application 217 receives arequest for a file 218, server application 217 retrieves correspondingfile 218 from system disk 206 and transmits the file 218 to an endpointdevice 115 or a content server 110 via network 105. Files 218 include aplurality of digital visual content items, such as videos and stillimages. In addition, files 218 may include metadata associated with suchvisual content items, user/subscriber data, etc. Files 218 that includevisual content item metadata and/or user/subscriber data may be employedto facilitate the overall functionality of network infrastructure 100.In alternative embodiments, some or all of files 218 may instead bestored in a control server 120, or in any other technically feasiblelocation within network infrastructure 100.

FIG. 3 is a block diagram of a control server 120 that may beimplemented in conjunction with the network infrastructure 100 of FIG.1, according to various embodiments. As shown, the control server 120includes, without limitation, a central processing unit (CPU) 304, asystem disk 306, an input/output (I/O) devices interface 308, a networkinterface 310, an interconnect 312, and a system memory 314.

CPU 304 is configured to retrieve and execute programming instructions,such as control application 317, stored in system memory 314. Similarly,CPU 304 is configured to store application data (e.g., softwarelibraries) and retrieve application data from system memory 314 and adatabase 318 stored in system disk 306.

Interconnect 312 is configured to facilitate transmission of databetween CPU 304, system disk 306, I/O devices interface 308, networkinterface 310, and system memory 314. I/O devices interface 308 isconfigured to transmit input data and output data between I/O devices316 and CPU 304 via interconnect 312. System disk 306 may include one ormore hard disk drives, solid state storage devices, and the like. Systemdisk 206 is configured to store a database 318 of information associatedwith content servers 110, cloud services 130, and files 218.

System memory 314 includes a control application 317 configured toaccess information stored in database 318 and process the information todetermine the manner in which specific files 218 will be replicatedacross content servers 110 included in the network infrastructure 100.Control application 317 may further be configured to receive and analyzeperformance characteristics associated with one or more of contentservers 110 and/or endpoint devices 115. As noted above, in someembodiments, metadata associated with such visual content items, and/oruser/subscriber data may be stored in database 318 rather than in files218 stored in content servers 110.

FIG. 4 is a block diagram of an endpoint device that may be implementedin conjunction with the network infrastructure of FIG. 1, according tovarious embodiments. As shown, endpoint device 115 may include, withoutlimitation, a CPU 410, a graphics subsystem 412, an I/O device interface414, a mass storage unit 416, a network interface 418, an interconnect422, and a memory subsystem 430.

In some embodiments, CPU 410 is configured to retrieve and executeprogramming instructions stored in memory subsystem 430. Similarly, CPU410 is configured to store and retrieve application data (e.g., softwarelibraries) residing in memory subsystem 430. Interconnect 422 isconfigured to facilitate transmission of data, such as programminginstructions and application data, between CPU 410, graphics subsystem412, I/O devices interface 414, mass storage 416, network interface 418,and memory subsystem 430.

In some embodiments, graphics subsystem 412 is configured to generateframes of video data and transmit the frames of video data to displaydevice 460. In some embodiments, graphics subsystem 412 may beintegrated into an integrated circuit, along with CPU 410. Displaydevice 450 may comprise any technically feasible means for generating animage for display. For example, display device 460 may be fabricatedusing liquid crystal display (LCD) technology, cathode-ray technology,and light-emitting diode (LED) display technology. An input/output (I/O)device interface 414 is configured to receive input data from user I/Odevices 462 and transmit the input data to CPU 410 via interconnect 422.For example, user I/O devices 462 may comprise one of more buttons, akeyboard, and a mouse or other pointing device. I/O device interface 414also includes an audio output unit configured to generate an electricalaudio output signal. User I/O devices 462 includes a speaker configuredto generate an acoustic output in response to the electrical audiooutput signal. In alternative embodiments, display device 460 mayinclude the speaker. Examples of suitable devices known in the art thatcan display video frames and generate an acoustic output includetelevisions, smartphones, smartwatches, electronic tablets, and thelike.

A mass storage unit 416, such as a hard disk drive or flash memorystorage drive, is configured to store non-volatile data. A networkinterface 418 is configured to transmit and receive packets of data vianetwork 105. In some embodiments, network interface 418 is configured tocommunicate using the well-known Ethernet standard. Network interface418 is coupled to CPU 410 via interconnect 422.

In some embodiments, memory subsystem 430 includes programminginstructions and application data that comprise an operating system 432,a user interface 434, and a playback application 436. Operating system432 performs system management functions such as managing hardwaredevices including network interface 418, mass storage unit 416, I/Odevice interface 414, and graphics subsystem 412. Operating system 432also provides process and memory management models for user interface434 and playback application 436. User interface 434, such as a windowand object metaphor, provides a mechanism for user interaction withendpoint device 108. Persons skilled in the art will recognize thevarious operating systems and user interfaces that are well-known in theart and suitable for incorporation into endpoint device 108.

In some embodiments, playback application 436 is configured to requestand receive content 440 from content server 105 via network interface418. Content 440 includes encoded frames of video and/or audio dataassociated with a media title. Those frames of video and/or audio aregenerally organized into “segments” that can be played back to the user.A given segment could be, for example, an introductory portion of aserial media title, a portion of a story arc associated with a branchingnarrative media title, a specific scene within a given media title, or acredit sequence associated with a given media title, among others.Playback application 436 is configured to interpret content 440 andpresent the content via display device 460 and/or user I/O devices 462.In so doing, playback application 436 may generate frames of video databased on content 440 and then transmit those frames of video data to aplatform player associated with endpoint device 115. In response, theplatform player causes display device 460 to output the frames of videodata. In one embodiment, operating system 432 includes one or moreplatform players.

In one embodiment, playback application 436 is configured to executeinstructions for displaying specific segments of content 440 based ontopology 450 and based on user input. Topology 450 sets forth a directedgraph of possible transitions between segments and constrains playbackof those segments to occur based on those possible transitions. Duringplayback, the user can provide input to endpoint device 115 to causeplayback to proceed along a desired set of segments, so long as thetransitions between those segments are allowed based on topology 450.

For example, content 440 could include a set of narrative componentsthat can be assembled based on user input to generate different storyarcs. Each narrative component could include a media segment that relaysa portion of a story. During playback, the user can select between anavailable set of options defined via topology 450 to cause playback toproceed to a desired segment. In response, playback application 436initiates playback of a subsequent media segment that is associated withthe selected option. By selecting different options, the user canexperience different story arcs.

In various other embodiments, playback can proceed along a particularpath for various reasons other than the receipt of user input. Forexample, lack of user input could cause a default path to be followed.State established earlier during playback might cause the availablepaths to be reduced to less than those available in topology 450,including reduction to a single path, where the existence of a branchpoint would be hidden from the user. In other use-cases outside of abranching narrative, different paths may correspond to differentlocalized versions of content 440 and so the paths for a given user maybe pre-determined based on language settings.

In various embodiments, playback application 436 analyzes content 440and topology 450 during playback in order to effect smooth transitionsbetween segments. In so doing, playback application 436 performs varioustechniques to reduce the playback of extraneous audio that can beintroduced via conventional encoding procedures, as discussed above.These techniques are described in greater detail below in conjunctionwith FIGS. 5A-13.

Encoding Audio Frames that Overlap Segment Boundaries

FIG. 5A illustrates how the content of FIG. 4 is organized in memory,according to various embodiments. As referred to herein, the term “inmemory” generally refers to the physical arrangement of data associatedwith media segments that potentially reside, at least in part, in avariety of different physical locations, including one or more fileswithin client memory, server memory, or elsewhere. As shown, content 440includes video segments 500 and audio frames 510 organized into segments520. Video segments 500 include video segments A, B, and C. Audio frames510 include audio frames 0, 1, 2, 3, 4, 5, and 6. Video segment Acorresponds to audio frames 0, 1, and 2. Video segment B corresponds toaudio frames 2, 3, and 4. Video segment C corresponds to audio frames 4,5, and 6. Each video segment 500 includes one or more video framesencoded with a particular video encoding standard. Each audio frame 510includes (or is derived from) one or more encoded audio samples encodedwith a particular audio encoding standard. Persons skilled in the artwill recognize that the number and lengths video segments and audioframes shown in FIG. 5A, among other places, are provided forillustrative purposes and are not drawn to scale.

As discussed above, a given segment can include video content as well ascorresponding audio content. One or more boundaries of a given segmentare defined based on the video portion of the segment. For example, asegment boundary 522(0) between segments 520(0) and 520(1) could bedefined based on the last video frame of video segment A and/or thefirst video frame of video segment B. Similarly, a segment boundary522(1) between segments 520(1) and 520(2) could be defined based on thelast video frame of video segment B and/or the first video frame ofvideo segment C.

Content 440 is generated during an encoding procedure that includes avideo encoding portion and an audio encoding portion. During the videoencoding portion of the encoding procedure, a video encoder processesraw video frames to generate the video frames within video segments 500.During an audio encoding portion of the encoding procedure, an audioencoder processes raw audio samples to generate audio frames 510associated with those video segments 500.

In various configurations, the video encoder and the audio encodergenerate video frames and audio frames that have different durations andtherefore have frame boundaries that sometimes do not align with oneanother. For example, when encoding a set of M raw audio samplesassociated with a given segment 520, the audio encoder could beconfigured to group together N raw audio samples to generate each audioframe, where M and N are non-negative integers. However, if N is not amultiple of M, then the audio encoder groups together some raw audiosamples from the end of that segment and some raw audio samples from thebeginning of a physically-subsequent segment in memory. As a result,certain audio frames overlap segment boundaries. For example, audioframe 2 overlaps segment boundary 522(0) and audio frame 4 overlapssegment boundary 522(1).

When playback proceeds across segments in accordance with the physicallayout of those segments in memory, audio frames that overlap segmentboundaries are decoded and played in conjunction with transitionsbetween those segments. Playback does not always proceed across segmentsin accordance with the physical layout of those segments in memory,though. In some cases, playback proceeds based on topology 450, asdescribed in greater detail below in conjunction with FIG. 5B.

FIG. 5B illustrates how the topology of FIG. 4 defines different logicalsequences of segments, according to various embodiments. As shown,topology 450 defines two different logical sequences that constrain theplayback of content 440. Logical sequence 552 allows segment 520(0)(including video segment A and corresponding audio) to be played backfollowed by segment 520(1) (including video segment B and correspondingaudio). Logical sequence 554 allows segment 520(0) (including videosegment A and corresponding audio) to be played back followed by segment520(2) (including video segment C and corresponding audio). Playback canproceed according to either logical sequence based on user input, amongother factors.

Segment 520(0) immediately precedes segment 520(1) in topology 440, asis shown, and segment 520(0) also immediately precedes segment 520(1) inmemory, as shown in FIG. 5A. Accordingly, the logical sequence ofsegments 520(0) and 520(1) in topology 440 may be considered consistentwith the physical sequence of segments 520(0) and 520(1) in memory.Segment 520(0) also immediately precedes segment 520(2) in topology 440,as is shown, but segment 520(0) does not immediately precede segment520(2) in memory, as shown in FIG. 5A. Accordingly, the logical sequenceof segments 520(0) and 520(2) in topology 440 may be consideredinconsistent with the physical sequence of segments 520(0) and 520(2) inmemory. Playback across logical sequences 552 and 554 differs based onwhether the logical sequence of the associated segments is consistentwith the physical sequence of those segments, as described in greaterdetail below in conjunction with FIGS. 6A-6B.

Identifying Extraneous Audio Data Associated with Segment Boundaries

FIG. 6A illustrates a logical sequence of segments that is consistentwith a physical sequence of those segments in memory, according tovarious embodiments. As shown, playback pathway 552 includes atransition 600 between segments 520(0) and 520(1). During playback,playback application 436 plays video segment A in conjunction with audioframes 0, 1, and 2 followed by video segment B in conjunction with audioframes 2, 3, and 4. Although audio frame 2 overlaps the physicalboundary between segments 520(0) and 520(1), audio frame 2 needs to beplayed in conjunction with both the last video frame of A and the firstvideo frame of B and so audio playback can proceed seamlessly acrosstransition 600 between segments 520(0) and 520(1). Audio playback doesnot proceed in this manner when the logical sequence of segments isinconsistent with the physical sequence of those segments in memory,though.

FIG. 6B illustrates a logical sequence of segments that is inconsistentwith a physical sequence of those segments in memory, according tovarious embodiments. As shown, playback pathway 554 includes atransition 610 between segments 520(0) and 520(2). Transition 610 maycorrespond to a “splice point” where segments 520(0) and 520(2) arespliced together. During playback, playback application 436 plays videosegment A in conjunction with audio frames 0, 1, and 2 followed by videosegment C in conjunction with audio frames 4, 5, and 6. Playbackapplication 436 decodes frame 2 in conjunction with the last video frameof video segment A and decodes frame 4 in conjunction with the firstvideo frame of video segment C.

However, a portion of audio frame 2 is derived from segment 520(1)because audio frame 2 overlaps the physical boundary between segments520(0) and 520(1) and segment 520(1) immediately follows segment 520(0)in memory. This portion of audio frame 2 is referred to as “extraneousaudio” and is shown in FIG. 6B as extraneous audio 612. The duration ofextraneous audio 612 can vary based on how much of audio frame 2overlaps the physical boundary between segments 520(0) and 520(1).Similarly, a portion of audio frame 4 is derived from segment 520(1)because audio frame 4 overlaps the physical boundary between segments520(1) and 520(2) and segment 520(1) immediately precedes segment 520(2)in memory. This portion of audio frame 4 is shown in FIG. 6B asextraneous audio 614. The duration of extraneous audio 614 can varybased on how much of audio frame 4 overlaps the physical boundarybetween segments 520(1) and 520(2).

Extraneous audio 612 and extraneous audio 614 can cause various issuesduring playback that can degrade the overall viewing experience. Inparticular, playing video with incorrect audio can be confusing to theuser. Additionally, that audio could have acoustic properties that areincompatible with other audio samples. For example, the audio samplesassociated with segments 520(0) and 520(2) could have a relatively lowamplitude, but extraneous audio 612 and/or 614 could have a relativelyhigh amplitude. In this scenario, playback application 436 mightabruptly play high-volume sounds, which may sound off to some users.Playback application 436 can implement several different techniques toanalyze and remove extraneous audio, as described in greater detailbelow in conjunction with FIGS. 7A-7C.

Selectively Dropping Audio Frames to Remove Extraneous Audio

FIG. 7A illustrates how the playback application of FIG. 4 selectivelydrops audio frames to reduce the playback of extraneous audio data,according to various embodiments. As mentioned above, the duration ofextraneous audio 612 and extraneous audio 614 can vary based on how muchof audio frames 2 and 4 overlap the associated physical boundariesbetween segments. Playback application 436 can analyze the durations ofextraneous audio 612 and extraneous audio 614 and then selectively dropeither audio frame 2 or audio frame 4 depending on which duration isgreater. In the example shown, the duration of extraneous audio 614exceeds that of extraneous audio 612 and so playback application 436drops audio frame 4. In doing so, playback application 436 introduces asmall gap G between the end of audio frame 2 and the beginning of audioframe 5.

The technique described in conjunction with FIG. 7A differs fromprevious approaches to mitigating extraneous audio because with previousapproaches, audio frames that include extraneous audio cannot beselectively dropped based on the duration of the extraneous audio.Importantly, because playback application 436 can account for theduration of extraneous audio when selectively dropping audio frames, theamount of extraneous audio that is played back to the user can beminimized. Playback application 436 can perform additional techniquesdepending on endpoint device capabilities, as described in greaterdetail below in conjunction with FIG. 7B.

FIG. 7B illustrates how the playback application of FIG. 4 drops andcrossfades audio frames to minimize the playback of extraneous audiodata, according to various embodiments. Some implementations of playbackapplication 436 support crossfading between audio frames and/or othertechniques for mitigating discontinuities. For example, browser-basedimplementations of playback application 436 that execute within thecontext of a web page would typically be able to perform crossfadeoperations. In the example shown, playback application 436 implementscrossfading techniques in conjunction with the selective frame droppingdiscussed above in conjunction with FIG. 7A. In particular, playbackapplication 436 selectively drops frame 4 because the duration ofextraneous audio 614 exceeds that of extraneous audio 612 and thenplayback application 436 performs crossfade 700 between a portion ofextraneous audio 612 and a portion of frame 5. In some instances,portion of extraneous audio 612 may have varying durations in order toadhere to one or more content delivery standards, such as the Hyper TextMarkup Language 5 (HTML5) standard.

The technique described in conjunction with FIG. 7B differs fromprevious approaches to mitigating extraneous audio for at least thereasons discussed above in conjunction with FIG. 7A. Additionally,because the duration of extraneous audio that is ultimately played backcan be minimized, the amount of crossfading needed can also beminimized, which reduces the amount of correct audio that is mixedtogether with extraneous audio. Playback application 436 can performanother technique that does not involve introducing gaps or performingcrossfades, as described in greater detail below in conjunction withFIG. 7C.

FIG. 7C illustrates how the playback application of FIG. 4 drops andshifts audio frames to minimize the playback of extraneous audio data,according to various embodiments. Some implementations of playbackapplication 436 support seamless audio transitions by shifting audioframes in time relative to corresponding video frames. This approach maybe known in the art as “frame seamless.” In the example shown, playbackapplication 436 implements frame seamless techniques in conjunction withthe selective frame dropping discussed above in conjunction with FIG.7A.

In particular, playback application 436 determines that the duration ofextraneous audio 614 exceeds that of extraneous audio 612 and, inresponse, drops frame 4. Playback application 436 then performs shift710 with frames 5 and 6 to allow a seamless audio transition from audioframe 2 to audio frame 5. In performing this technique, playbackapplication 436 generally maintains an audio/visual (AV) synchronization(sync) error that reflects a time offset between video playback andaudio playback. Playback application 436 can drop additional frames thatinclude extraneous audio, such as frame 2, if the maximum AV sync errorwould not be exceeded. In various embodiments, playback application 436may maintain a maximum AV sync error that is equal to the duration ofone audio frame or one video frame. In other embodiments, playbackapplication 436 maintains a maximum AV sync error that is equal to anyfractional amount of the duration of one audio frame or one video frame.

As a general matter, in various embodiments, there may be both a minimumand a maximum bound on AV sync error, where the AV sync error is definedto be how far the audio is behind the video (where negative numbersimply that the audio is ahead of the video). In some cases, the maximumAV sync error can be one audio frame. In other cases, the maximum AVsync error can be any fractional amount of the duration of an audioframe. The minimum AV sync error may be equal to the maximum AV syncerror minus one audio frame, however it could be set less than this andthat could offer more opportunities to drop extraneous audio.

The technique described in conjunction with FIG. 7C differs fromprevious approaches to mitigating extraneous audio for at least thereasons discussed above in conjunction with FIG. 7A. Additionally, thedisclosed technique allows audio frames to be played back seamlesslyacross transitions between segments, thereby avoiding gaps andcrossfades.

Referring generally to FIGS. 7A-7C, playback application 436 canimplement the above techniques depending on endpoint device capabilitiesand/or configuration. The above techniques are described in greaterdetail below in conjunction with FIG. 8.

FIG. 8 is a flow diagram of method steps for reducing the playback ofextraneous audio data, according to various embodiments. Although themethod steps are described in conjunction with the systems of FIGS.1-7C, persons skilled in the art will understand that any systemconfigured to perform the method steps, in any order, is within thescope of the present embodiments.

As described below, playback application 436 performs various techniquesto selectively drop audio frames (potentially introducing a gap), asdiscussed in conjunction with steps 806, 808, and 810 of the method 800,crossfade audio frames, as discussed in conjunction with steps 812 and814 of the method 800, and shift audio frames, as discussed inconjunction with steps 816, 818, and 820 of the method 800. In someconfigurations, playback application 436 may be pre-configured toimplement just one of these techniques without performing the other twotechniques. For example, playback application 436 could be configured toonly perform steps 816, 818, and 820 to shift audio frames withoutperforming steps 806, 808, 810, 812, and 814 to drop and/or crossfadeaudio frames. The various techniques are presented here in conjunctionwith the method 800 to explain the range of techniques that playbackapplication 436 can be configured to implement. Any given technique,when implemented, may affect, or be affected by, other logic thatplayback application 436 implements to drop one or more audio frames.

In one embodiment, playback application 436 may implement a conventionaltechnique to first determine the number of audio frames to drop, eitherto leave a gap, an overlap for crossfading, or to modify the AV syncerror via frame shifting, and then subsequently perform a noveltechnique whereby the choice of which frames to drop is informed by thelogical topology of the media content and/or the duration of extraneousaudio. As shown, a method 800 begins at step 802, where playbackapplication 436 identifies a first portion of extraneous audio thatextends past a transition between segments. Extraneous audio generallyarises during playback of two segments when the logical sequencing ofthose segments in the relevant topology is not consistent with thephysical sequencing of those segments in memory. Playback application436 identifies the first portion of extraneous audio by determining thatthe last audio frame of a first one of the two segments overlaps aphysical boundary with a different segment.

At step 804, playback application 436 identifies a second portion ofextraneous audio that extends past the transition between segments.Extraneous audio can extend past a given transition by extending afterthe transition, as with extraneous audio 612, or extending before thetransition, as with extraneous audio 614. Playback application 436identifies the second portion of extraneous audio by determining thatthe first audio frame of a second one of the two segments overlaps aphysical boundary with a different segment.

At step 806, playback application 436 determines whether the secondportion of extraneous audio has a greater duration than the firstportion of extraneous audio. If at step 806 playback application 436determines that the duration of the first portion of extraneous audioexceeds the duration of the second portion of extraneous audio, then themethod 800 proceeds to step 808.

At step 808, playback application 436 drops the audio frame thatincludes the second portion of extraneous audio. This approach differsfrom previous approaches that apply fixed rules to dropping audio framesthat do not depend on the duration of the portions of extraneous audio.For example, a conventional approach could always drop the first portionof extraneous audio independently of the length of the two portions ofextraneous audio. In some cases, dropping the audio frame that includesthe second portion of extraneous audio introduces a small gap, asdiscussed above in conjunction with FIG. 7A. The method 800 thenproceeds to step 812.

If at step 806 playback application 436 determines that the firstportion of extraneous audio exceeds the duration of the second portionof extraneous audio, then the method 800 proceeds to step 810. At step810, playback application 436 drops the audio frame that includes thefirst portion of extraneous audio. In some cases, dropping the audioframe that includes the first portion of extraneous audio introduces asmall gap, as discussed above in conjunction with FIG. 7A. In thismanner, playback application 436 can selectively drop audio frames thatinclude extraneous audio depending on the duration of that extraneousaudio. The method 800 then proceeds to step 812.

At step 812, playback application 436 determines whether crossfadeoperations are available. Playback application 436 can perform crossfadeoperations and/or other operations for mitigating discontinuitiesdepending on the capabilities of endpoint device 115 and/or the specificimplementation of playback application 436. For example, if endpointdevice 115 is a computing device that executes playback application 436within a web browser, then playback application 436 could be capable ofperforming crossfade operations. If at step 812 playback application 436determines that audio frames can be crossfaded, then the method proceedsto step 814. In one embodiment, playback application 436 may implement aconventional rule-based approach to drop both portions of extraneousaudio, subsequently allowing one or more audio frames to be shifted, asdescribed below.

At step 814, playback application 436 crossfades any overlapping audioframes that remain after dropping an audio frame in conjunction witheither step 808 or 810. In doing so, playback application 436 decodesthe overlapping audio samples to generate two sets of audio samples. Theaudio samples could be, for example, pulse-code modulation (PCM)samples. Playback application 436 then computes a weighted sum of thefirst and second sets of audio samples, where the weighting applied tocorresponding pairs of samples changes across the duration of thecrossfade. An example of how playback application 436 could perform step814 is described above in conjunction with FIG. 7B.

If at step 812 playback application 436 determines that audio framescannot be crossfaded, then the method proceeds to step 816. At step 816,playback application 436 determines whether any audio frames can beshifted relative to corresponding video frames. Playback application 436could, for example, analyze the current AV sync error and determine thatthe current AV sync error is sufficiently below a maximum AV sync errorthat frame-shifting can be performed.

If at step 816 playback application 436 determines that audio frames canbe shifted relative to video frames, then the method 800 proceeds tostep 818. At step 818, playback application 436 shifts one or more audioframes associated with a second segment to play immediately after thefinal audio frame of a first segment, thereby implementing a seamlessaudio transition. An example of how playback application 436 couldperform step 816 is described above in conjunction with FIG. 7C.

If at step 816 playback application 436 determines that audio framescannot be shifted relative to video frames, then the method 800 proceedsto step 820. At step 820, playback application 436 plays back theremaining audio frames across the transition point between segments. Asmentioned above, dropping an audio frame in conjunction with either ofsteps 808 and 810 can introduce a small gap. Accordingly, in performingstep 820, playback application 436 can sometimes play audio across thetransition with a small gap.

In performing the method 800, playback application 436 can reduce theamount of extraneous audio that is output during playback of content 440based on an analysis of the extraneous audio. In addition, playbackapplication 436 can analyze the extraneous audio in conjunction withtopology 440 in order to perform other techniques for mitigatingplayback of extraneous audio, as described in greater detail below inconjunction with FIGS. 9-12B.

Topology-Driven Reduction of Extraneous Audio

FIG. 9 illustrates how the playback application of FIG. 4 reducesextraneous audio data based on a media title topology, according tovarious embodiments. As shown, during playback of logical sequence 454of topology 450, audio frames 2 and 4 are decoded in conjunction withtransition 610 to produce extraneous audio 612 and 614, as previouslydiscussed. Extraneous audio 612 corresponds to a beginning portion ofsegment 520(1), as depicted with regard to topology 450, whileextraneous audio 614 corresponds to an ending portion of segment 520(2),as also depicted with regard to topology 450. Segment 520(1) generallyresides physically before segment 520(2).

During playback of logical sequence 454, playback application 436 cananalyze topology 450 and determine whether the audio samples associatedwith extraneous audio 612 and/or extraneous audio 614 have similar audiowaveforms compared to the audio samples that are supposed to be playedin conjunction with video segments A and/or C. If the audio waveformassociated with a given portion of extraneous audio is significantlydifferent from the audio samples corresponding to video segments A andC, then playback application 436 can drop the audio frame that includesthat portion of EA. If the audio waveform associated with the givenportion of extraneous audio is not significantly different from theaudio samples corresponding to video segments A and C, then playbackapplication 436 need not drop the corresponding audio frame becauseplayback of the portion of extraneous audio may not be noticeable tousers. With this approach, playback application 436 can selectivelyidentify extraneous audio that diverges acoustically from other audiothat is to be played in conjunction with a transition between segments.

Playback application 436 can determine whether any two sets of samplescorrespond to similar or different audio waveforms using a variety oftechniques. In one embodiment, segments 520 that share a logicalpredecessor in topology 450 are subject to an authoring constraint thataudio samples within corresponding regions of those segments havesimilar audio waveforms. For example, segments 520(1) and 520(2) sharesegment 520(0) as a logical predecessor and could be subject to anauthoring constraint that audio samples within an interval T_(i) havesimilar audio waveforms. Playback application 436 may determine that twosets of samples do not have significantly different audio waveforms upondetermining that those two sets of samples reside in correspondingregions of two respective segments that share a logical predecessor. Inanother embodiment, playback application 436 may determine that two setsof samples have similar audio waveforms by directly comparing the twosets of samples to one another.

In the example shown, playback application 436 analyzes topology 450 anddetermines that the audio samples included in initial interval T_(i) ofboth segments 520(1) and 520(2) have similar audio waveforms. Based onthis determination, playback application 436 determines that extraneousaudio 612 includes audio samples that do not diverge acoustically fromthe audio samples that are supposed to be played in conjunction withvideo segment C. Playback application 436 can then refrain from droppingextraneous audio 612 because playback of extraneous audio 612 may not benoticeable to users.

Conversely, playback application 436 can analyze topology 450 anddetermine that extraneous audio 614, which corresponds to the endportion of segment 530(1), likely includes audio samples that divergeacoustically from the audio samples that are meant to be played inconjunction with video segment C. Among other things, the audio samplesassociated with extraneous audio 614 do not immediately follow the audiosamples associated with audio frame 2 and do not immediately precede theaudio samples associated with audio frame 4 (from a logicalperspective). Accordingly, the presence of these samples conflicts withthe logical sequence of segments set forth in topology 450. Based onthis determination, playback application 436 can drop audio frame 4.

With the above approach, playback application 436 can identify andreduce extraneous audio that could be noticeable users and keepextraneous audio that may not be noticeable to users. This approach canbe combined with any of the other techniques described thus far. Forexample, if both portions of extraneous audio include audio samples withacoustic incompatibilities, then playback application 436 can reduce thelonger of the two portions, as discussed above in conjunction with FIG.7A. Playback application 436 can then crossfade the remaining audiosamples, if available, as discussed above in conjunction with FIG. 7B.Playback application 436 can alternatively shift the remaining audiosamples to facilitate a seamless audio transition, as discussed above inconjunction with FIG. 7C.

In certain situations, playback application 436 determines that one ormore frames should be dropped to reduce portions of extraneous audio butalso determines that those frames cannot be dropped without causing theAV sync error to exceed a maximum AV sync error. In such situations,playback application 436 can implement one or more the techniquesdescribed below in conjunction with FIGS. 10A-10B.

Mitigating Extraneous Audio without Exceeding AV Sync Error

FIG. 10A illustrates how the playback application of FIG. 4 duplicatesan audio frame to avoid playing back extraneous audio data, according tovarious embodiments. In the example shown, playback application 436analyzes audio frames 2 and 4 and determines that audio frame 2 shouldbe dropped and the remaining audio frames should be shifted to implementa seamless audio transition. For example, playback application 436 couldbe configured to implement the techniques described above in conjunctionwith FIGS. 7A and 7C to selectively drop audio frames and then shift theremaining audio frames. However, playback application 436 could alsodetermine that the current AV sync error cannot be increased withoutexceeding the maximum AV sync error, thereby disallowing any frames frombeing shifted.

In this situation, playback application 436 can replace audio frame 2with audio frame 1, thereby eliminating extraneous audio 612 but alsomaintaining the current AV sync error. Playback application 436 canimplement a similar technique to replace audio frame 4 with audio frame5, if needed, thereby eliminating extraneous audio 614 but alsomaintaining the current AV sync error. The remaining audio frames can becrossfaded at transition 610, if crossfading is available. An advantageof this approach is that the duplicated audio frames include audiosamples that are not acoustically divergent from other audio samplesplayed in conjunction with transition 610, thereby reducing the playbackof incorrect audio and/or audio artifacts.

FIG. 10B illustrates how the playback application of FIG. 4 moves atransition between audio frames to avoid playing back extraneous audiodata, according to various embodiments. In the example shown, playbackapplication 436 analyzes audio frame 2 and determines, based on topology450, that extraneous audio 612 is not acoustically divergent and neednot be removed. Playback application 436 also analyzes audio frame 4 anddetermines that extraneous audio 614 is acoustically divergent butcannot be removed without causing the maximum AV sync error to beexceeded.

In this situation, playback application 436 can play an additional audioframe that physically follows audio frame 2 in memory (audio frame 3),effectively implementing an audio transition 1000 that occurs after thetransition between video segments A and C. Audio frame 3 includes audiosamples that are wholly associated with video segment B, although theseaudio samples are not acoustically divergent from those associated withthe beginning portion of video segment C. Audio frame 4 can be droppedand audio frames 3 and 5 can be crossfaded (if crossfading isavailable). An advantage of this approach is that extraneous audio canbe removed without disrupting the current AV sync error.

Referring generally to FIGS. 5A-10B, the disclosed techniques can beimplemented with any technically feasible content and any correspondingtopology. FIGS. 11A-12B illustrate how the foregoing techniques areapplicable to a different type of topology than that discussed above inconjunction with FIGS. 5A-10B.

Mitigating Extraneous Audio in a Divergent Topology

FIG. 11A illustrates how the content of FIG. 4 can be organized inmemory, according to various other embodiments. As shown, content 440′includes video segments 1100 and audio frames 1110 organized intosegments 1120. Video segments 1100 include video segments X, Y, and Z.Audio frames 1110 include audio frames 10, 11, 12, 13, 14, 15, and 16.Video segment X corresponds to audio frames 10, 11, and 12. Videosegment Y corresponds to audio frames 12, 13, and 14. Video segment Zcorresponds to audio frames 14, 15, and 16. Each video segment 1100includes one or more video frames encoded with a particular videoencoding standard. Each audio frame 1110 includes one or more encodedaudio samples encoded with a particular audio encoding standard. Segmentboundaries 1122 demarcate the end of one segment and the start ofanother. Audio frames 12 and 14 overlap segment boundaries 1122(0) and1122(1), respectively.

FIG. 11B is an alternative illustration of the topology of FIG. 4,according to various other embodiments. As shown, topology 450′ definestwo different logical sequences that constrain the playback of content440′. Logical sequence 1152 allows segment 1120(0) (including videosegment X and corresponding audio) to be played back followed by segment1120(2) (including video segment Y and corresponding audio). Logicalsequence 1154 allows segment 1120(1) (including video segment Y andcorresponding audio) to be played back followed by segment 1120(2)(including video segment Z and corresponding audio).

Segment 1120(0) immediately precedes segment 1120(2) in topology 450′,as is shown, but segment 1120(0) does not immediately precede segment1120(2) in memory, as shown in FIG. 11A. Accordingly, the logicalsequence of segments 1120(0) and 1120(2) in topology 450′ isinconsistent with the physical sequence of segments 1120(0) and 1120(2)in memory. Segment 1120(1) immediately precedes segment 1120(2) intopology 450′, as is shown, and segment 1120(1) also immediatelyprecedes segment 1120(2) in memory, as shown in FIG. 11A. Accordingly,the logical sequence of segments 1120(1) and 1120(2) in topology 450′ isconsistent with the physical sequence of segments 1120(1) and 1120(2) inmemory. Playback across logical sequences 1152 and 1154 differs based onwhether the logical sequence of the associated segments is consistentwith the physical sequence of those segments, as described in greaterdetail below in conjunction with FIGS. 12A-12B.

FIG. 12A illustrates a logical sequence of segments within the topologyof FIG. 11B, according to various embodiments. As shown, logicalsequence 1152 includes a transition 1200 between segments 1120(0) and1120(2). During playback, playback application 436 plays video segment Xin conjunction with audio frames 10, 11, and 12 followed by videosegment Z in conjunction with audio frames 14, 15, and 16. Audio frame12 overlaps segment boundary 1122(0) and includes audio samples derivedfrom the beginning portion of segment 1120(1), shown as extraneous audio1212. Audio frame 14 overlaps segment boundary 1122(1) and includesaudio samples derived from the end portion of segment 1120(1), shown asextraneous audio 1214.

Playback application 436 can implement any of the techniques describedabove in conjunction with FIGS. 5A-10B to mitigate or reduce extraneousaudio 1212 and 1214. For example, playback application 436 couldselectively drop the longer of the two portions of extraneous audio andintroduce a gap, implement a crossfade, or shift remaining audio frames,as described above in conjunction with FIGS. 7A-7C. Playback application436 could also analyze topology 450′ and determine whether any ofextraneous audio 1112 and/or extraneous audio 1114 include audio samplesthat are acoustically divergent from the audio samples that are supposedto be played in conjunction with video segments X and/or Z. Playbackapplication 436 could then selectively remove either or both ofextraneous audio 1112 and 1114, as described above in conjunction withFIG. 9. In one embodiment, segments 1120 may be subject to an authoringconstraint that segments sharing a common logical successor, such assegments 1120(0) and 1120(1), should include a final portion of audiohaving relatively similar audio waveforms. Playback application 436could also duplicate one or more audio frames to replace either or bothof audio frames 12 and 14 and/or shift the audio transition point, asdescribed above in conjunction with FIGS. 10A-10B.

FIG. 12B illustrates another logical sequence of segments within thetopology of FIG. 11B, according to various embodiments. As shown,logical sequence 1154 includes a transition 1210 between segments1120(1) and 1120(2). During playback, playback application 436 playsvideo segment Y in conjunction with audio frames 12, 13, and 14 followedby video segment Z in conjunction with audio frames 14, 15, and 16.Audio frame 14 overlaps segment boundary 1120(1). However, because audioframe 14 is supposed to be played in conjunction with the last videoframe of video segment Y and the first video frame of video segment Z,audio playback can proceed normally across transition 1210.

Referring generally to FIGS. 5A-12B, persons skilled in the art willunderstand how the techniques described in conjunction with theseFigures can be applied to any technically feasible topology, includinglogically divergent topologies, logically convergent topologies, and anycombination thereof.

FIG. 13 is a flow diagram of method steps for reducing the playback ofextraneous audio data, according to various other embodiments. Althoughthe method steps are described in conjunction with the systems of FIGS.1-12B, persons skilled in the art will understand that any systemconfigured to perform the method steps, in any order, is within thescope of the present embodiments.

As shown, a method 1300 begins at step 1302, where playback application436 identifies a first portion of extraneous audio and a second portionof extraneous audio. The first and second portions of extraneous audiocan be included in segments associated with a media title. A givensegment can include encoded video frames and/or audio frames. Playbackapplication 436 can identify a portion of extraneous audio associatedwith a given audio frame by determining that the given audio frameoverlaps a physical boundary between segments that reside sequentiallyin memory. Audio frames that overlap segment boundaries can be generatedwhen an audio encoder groups together some raw audio samples associatedwith the end of a segment with other raw audio samples associated withthe beginning of a segment.

At step 1304, playback application 436 determines which portions ofextraneous audio conflict with a topology corresponding to the mediatitle. The topology defines a set of segments and a set of allowabletransitions between those segments. Playback of the media title canproceed between segments based on those allowable transitions and basedon user input, among other things. During playback across a logicalsequence of segments, playback application 436 can determine that agiven portion of extraneous audio conflicts with the topology bydetermining that the given portion of extraneous audio should not beplayed in conjunction with that logical sequence of segments. The givenportion of extraneous audio can include audio samples that areacoustically divergent from other audio samples that are supposed to beplayed in conjunction with the logical sequence of segments. As such,playback of the given portion of extraneous audio may sound incorrect.

At step 1306, playback application 436 determines whether any audioframes can be shifted relative to video frames. Playback application 436maintains an accumulated AV sync error that defines the current temporaloffset between video frames and audio frames. Playback application 436can shift audio frames relative to video frames when doing so does notcause the AV sync error to exceed a maximum AV sync error. If at step1306 playback application 436 determines that audio frames can beshifted relative to video frames, then the method 1300 proceeds to step1308.

At step 1308, playback application 436 determines whether only one ofthe two portions of extraneous audio identified at step 1302 conflictswith the topology of the media title. In some cases, the varioussegments associated with the media title may be subject to an authoringconstraint that segments having a common logical predecessor (orsuccessor) in the topology should include an initial (or final) portionof audio having relatively similar audio waveforms. In some instancesthis approach allows one of the two portions of extraneous audio to beretained and the audio frame that includes the other portion ofextraneous audio to be dropped.

If at step 1308 playback application 436 determines that only oneportion of extraneous audio conflicts with the media title topology,then the method 1300 proceeds to step 1310. At step 1310, playbackapplication 436 drops the audio frame that includes the one portion ofextraneous audio. Because playback application 436 determines at step1306 that audio frames can be shifted relative to video frames, theremaining audio frames can be shifted to implement a seamless audiotransition. This technique is described above in conjunction with FIG.9.

If at step 1308 playback application 436 determines that both portionsof extraneous audio conflicts with the media title topology, then themethod 1300 proceeds to step 1312. At step 1312, playback application436 drops the audio frame that includes the longer portion of extraneousaudio, in like fashion as described above in conjunction with FIG. 7A.Playback application 436 can then shift the remaining audio frames toimplement a seamless audio transition.

Returning to step 1306, if playback application 436 determines at step1306 that audio frames cannot be shifted relative to video frames, thenthe method 1300 proceeds to step 1314. At step 1314, playbackapplication 436 determines whether an audio transition point should bemoved ahead or behind a transition between corresponding video segments.If at step 1314 playback application 436 determines that the audiotransition point should not be moved, then the method 1300 proceeds tostep 1316.

At step 1316, playback application 436 replaces one or more audio framesthat include extraneous audio with one or more adjacent audio frames.These adjacent audio frames generally include audio samples that do notconflict with the topology and can therefore be used as viablereplacements for audio frames that should not be played. This techniqueis described above in conjunction with FIG. 10A.

If at step 1314 playback application 436 determines that the audiotransition point should be moved, then the method 1300 proceeds to step1318. At step 1318, playback application 436 moves the audio transitionpoint past the extraneous audio that does not conflict with the mediatitle topology. In doing so, playback application 436 causes anadditional audio frame to be played that resides physically subsequentto the audio frame that includes the portion of extraneous audio.Playback application 436 can then drop the frame that includes theconflicting extraneous audio without disrupting the AV sync error. Thistechnique is described above in conjunction with FIG. 10B.

Persons skilled in the art will understand that the method 1300 isapplicable to any technically feasible topology, including divergenttopologies such as topology 450 discussed above in conjunction withFIGS. 5B-10B as well as convergent topologies such as topology 450′discussed above in conjunction with FIGS. 11B-12B. The disclosedtechniques can advantageously be applied to reduce playback ofextraneous audio that may be disruptive to the viewing experience. Inaddition to these techniques, preprocessing engine 132 of FIG. 1 canmodify raw audio samples associated with segments 520 in orderpreemptively mitigate extraneous audio, as described in greater detailbelow in conjunction with FIGS. 14A-15.

Modifying Raw Audio Samples to Remove Extraneous Audio

FIG. 14A illustrates raw media content that is encoded to generate thecontent of FIG. 4, according to various embodiments. As shown, rawcontent 1440 includes raw video frames 1400 and raw audio samples 1410.Raw video frames 1400 include raw video segments rA, rB, and rC. Rawvideo segments rA, rB, and rC can be encoded to generate video segmentsA, B, and C of FIG. 5A. Raw video segments rA, rB, and rC are organizedinto segments 520(0), 520(1), and 520(2) similarly to how video segmentsA, B, and C of FIG. 5A are organized into segments 520(0), 520(1), and520(2). Segment boundaries 522(0) and 522(1) demarcate the ends of rawvideo segments rA and rB and the beginnings of raw video segments rB andrC, respectively.

Raw audio samples 1410 can be grouped together and encoded to generateaudio frames 510 of FIG. 5A. As discussed, depending on audio encodersettings, groups of raw audio samples 1410 can be generated that overlapsegment boundaries. In particular, group 1412 of raw audio samplesoverlaps segment boundary 522(0) and group 1414 of raw audio samplesoverlaps segment boundary 522(1). During audio encoding, groups 1412 and1414 of raw audio samples can be encoded to generate audio frames 2 and4, respectively, shown in FIG. 5A. Extraneous audio can be played whenaudio frames of specific segments 520 are logically sequenced accordingto topology 450 but are not physically sequenced in memory, as is shownin FIG. 14B.

Referring now to FIG. 14B, as shown, segments 520(0) and 520(1) arelogically sequenced according to topology 450 and also physicallysequenced in memory. Accordingly, audio frames associated with atransition between those segments typically do not include extraneousaudio. Segments 520(0) and 520(2), by contrast, are logically sequencedaccording to topology 450 but not physically sequenced in memory.Consequently, audio frames associated with a transition between thosesegments may include extraneous audio. Referring generally to FIGS. 14Aand 14B, any two segments 530 that are logically sequenced but notphysically sequenced can lead to the occurrence of extraneous audio forthe reasons discussed above in conjunction with FIGS. 5A-6B.

Preprocessing engine 132 addresses this issue by analyzing raw audiosamples 1410 in conjunction with segment boundaries 1422 to identifygroups 1412 and 1414 of raw audio samples 1410 prior to encoding.Preprocessing engine 132 then analyzes topology 450 and identifies anysegments where the logically preceding segment is not the physicallypreceding segment. Audio samples that reside nearby in memory to suchsegments can potentially become extraneous audio when encoded into anaudio frame that overlaps a segment boundary. Preprocessing engine 132replaces these audio samples with other audio samples derived from thelogically preceding segment. Because the replacement audio samples arederived from the logically preceding segment, those audio samples maynot disrupt audio playback during transitions between the two segments.

In the example shown, preprocessing engine 132 determines that segment520(2) is logically preceded by segment 520(0) but not physicallypreceded by segment 520(0). Preprocessing engine 132 identifies subset1416 of raw audio samples that reside at the end of segment 520(1).Subset 1416 of raw audio samples can potentially be grouped togetherduring encoding into an audio frame that is played back in conjunctionwith the first video frame of segment 520(2). In other words, subset1416 can cause extraneous audio to be played in conjunction withincorrect video frames. Preprocessing engine 132 also identifies asubset 1418 of raw audio samples that reside at the end of segment520(0) and includes the same number of raw audio samples as subset 1416.Preprocessing engine 132 replaces subset 1416 with subset 1418.Subsequently, during encoding, group 1414 of raw audio samples can beencoded into an audio frame that overlaps a segment boundary but doesnot include raw audio samples associated with a non-logically precedingsegment. This approach facilitates a smooth audio transition betweensegments 520(0) and 520(2) and prevents the playback of extraneous audioassociated with the end of segment 520(1). In one embodiment, thisapproach may be specifically applicable to situations where the topologyis subject to the authoring constraint that the initial portions of anysegments that share a logical predecessor include relatively similaraudio waveforms.

An analogous technique to that described above can be applied toconvergent topologies, such as topology 450′ of FIG. 11B. Withconvergent topologies (or convergent portions of topologies),preprocessing engine 132 identifies segments where the logicallysubsequent segment in the topology is not the physically subsequentsegment in memory. Preprocessing engine 132 then replaces a subset ofsamples from the end of the identified segment with a correspondingsubset of samples from the beginning of the logically subsequentsegment. In one embodiment, this approach may be specifically applicableto situations where the convergent topology is subject to the authoringconstraint that the final portions of any segments that share a logicalsuccessor include relatively similar audio waveforms.

FIG. 15 is a flow diagram of method steps for modifying raw audiosamples to reduce the playback of extraneous audio data, according tovarious embodiments. Although the method steps are described inconjunction with the systems of FIGS. 1-14B, persons skilled in the artwill understand that any system configured to perform the method steps,in any order, is within the scope of the present embodiments.

As shown, a method 1500 begins at step 1502, where preprocessing engine132 of FIG. 1 analyzes the topology of a media title to determine afirst segment where the physically preceding segment is not thelogically preceding segment. For example, preprocessing engine 132 couldanalyze topology 450 and determine that segment 530(2) is logicallypreceded by segment 530(0) but physically preceded by segment 530(1) inmemory. This arrangement of segments can lead to audio frames thatoverlap segment boundaries and potentially lead to the playback ofextraneous audio.

At step 1502, preprocessing engine 132 analyzes the topology of themedia title to determine a second segment that logically precedes thefirst segment. For example, preprocessing engine 132 could analyzetopology 450 and determine that segment 530(0) logically precedessegment 530(2). The second segment may include raw audio samples thatcan be used to replace raw audio samples within the first segmentcorresponding to extraneous audio.

At step 1504, preprocessing engine 132 analyzes raw audio samplesassociated with the media title to determine N audio samples thatprecede the start of the first segment and are positioned for encodinginto a first audio frame associated with the first segment. These Nsamples generally reside just before a segment boundary that demarcatesthe start of the first segment. For example, preprocessing engine 132could determine subset 1416 of raw audio samples that reside just beforesegment boundary 522(1).

At step 1506, preprocessing engine 132 analyzes raw audio samplesassociated with the media title to determine the last N samplesassociated with the second segment. For example, processing engine 132could identify subset 1418 of raw audio samples that reside at the endof segment 520(0). These audio samples can be used to replace the audiosamples determined at step 1504.

At step 1508, preprocessing engine 132 modifies raw audio samplesassociated with the media title to replace the N audio samples thatprecede the start of the first segment with the last N samplesassociated with the second segment. Subsequently, during audio encoding,the audio encoder can generate an audio frame that overlaps the physicalboundary demarcating the start of the first segment withoutincorporating raw audio samples associated with the physically precedingsegment. Instead, the audio encoder generates the audio frame toincorporate the replacement samples from the logically precedingsegment. During playback, this audio frame can be decoded with reducedextraneous audio.

In one embodiment, at step 1508, instead of replacing the samplespreceding the start of the first segment preprocessing engine 132 mayinsert those N samples immediately before the first segment, whereinserting the N samples causes subsequent samples to be delayed in timeto accommodate the insertion of the N samples. This approach causes theaudio file to become longer than the corresponding video and thelocations and durations of all such inserted samples would becommunicated to the client so that the AV sync can be appropriatelymodified, depending on the part of the media that is being rendered. Theinsertion approach addresses one disadvantage of the replacementapproach, which is that although replacing audio samples removesextraneous audio for one logical transition, doing so may in some casesintroduce additional extraneous audio for another logical transition,depending on the topology and frame timing. While the replacementtechnique enables an improvement, this technique may in some cases notremove all extraneous audio issues. The insertion technique may be ableto address all such issues.

In sum, a playback application is configured to analyze audio framesassociated with transitions between segments within a media title toidentify one or more portions of extraneous audio. The playbackapplication is configured to analyze the one or more portions ofextraneous audio and then determine which of the one or morecorresponding audio frames should be dropped. In doing so, the playbackapplication can analyze a topology associated with the media title todetermine whether any specific portions of extraneous audio are to beplayed outside of a logical ordering of audio samples set forth in thetopology. These specific portions of extraneous audio are preferentiallyremoved.

In addition to the foregoing techniques, various embodiments include apreprocessing engine that modifies the raw audio samples associated withthe media title prior to encoding in order to avoid the occurrence ofextraneous audio. The preprocessing engine analyzes the topology of themedia title in conjunction with analyzing the physical layout of themedia title in memory. Based on these analyses, the preprocessing engineidentifies pairs of segments of the media title that are logicallysequenced according to the topology but not physically sequenced inmemory. The preprocessing engine analyzes a set of raw audio samplesassociated with one of the segments associated with a given pair ofsegments and then replaces a subset of those raw audio samples withanother subset of raw audio samples associated with the other segment inthe pair. In this manner, the preprocessing engine can preemptivelyavoid playback of extraneous audio.

At least one technological advantage of the disclosed techniquesrelative to the prior art is that portions of audio data that are notmeant to be played back in conjunction with video frames derived fromone or more segments of a media title can be more effectively reduced.Accordingly, situations where incorrect audio is output during playbackof a given segment can be largely avoided, thereby enhancing the overallquality of the user experience when streaming media titles. Anothertechnological advantage of the disclosed techniques relative to theprior art is that portions of audio data that are acoustically divergentfrom other portions of audio data can be more effectively reduced whenanalyzed relative to the logical topology of a given media title.Accordingly, situations where jarring and unexpected audio data areinadvertently output to the user can be largely avoided, therebyenhancing the overall streaming experience. Yet another technologicaladvantage of the disclosed techniques relative to the prior art is thatextraneous audio data can be preemptively removed from raw audio samplesand replaced with non-extraneous audio. Accordingly, the occurrence ofincorrect audio during playback of a given media title can be reduced,further improving the overall quality with which media titles can bestreamed. These technological advantages represent one or moretechnological advancements relative to prior art approaches.

1. Some embodiments include a computer-implemented method, comprisingidentifying a first portion of audio data that should not be played backin conjunction with a first sequence of video segments associated with amedia title, identifying a second portion of audio data that also shouldnot be played back in conjunction with the first sequence of videosegments, determining that the first portion of audio data has a greaterplayback duration than the second portion of audio data, and preventingthe first portion of audio data from being played back in conjunctionwith the first sequence of video segments.

2. The computer-implemented method of clause 1, wherein preventing thefirst portion of audio data from being played back in conjunction withthe first sequence of video segments comprises canceling one or moreplayback operations involving a first audio frame, wherein the firstaudio frame includes the first portion of audio data, and the firstaudio frame overlaps a physical boundary associated with a first videosegment included in the first sequence of segments and a second videosegment included in the first sequence of segments.

3. The computer-implemented method of any of clauses 1-2, furthercomprising performing a cross-fade operation between the second portionof audio data and a third portion of audio data that should be playedback in conjunction with the first sequence of video segments.

4. The computer-implemented method of any of clauses 1-3, furthercomprising performing a transition operation between the second portionof audio data and a third portion of audio data that should be playedback in conjunction with the first sequence of video segments.

5. The computer-implemented method of any of clauses 1-4, wherein alogical arrangement of video segments associated with the media titleindicates that the first sequence of video segments includes a firstvideo segment followed by a second video segment and that a secondsequence of video segments includes the first video segment followed bya third video segment.

6. The computer-implemented method of any of clauses 1-5, wherein aphysical arrangement of video segments associated with the media titleindicates that the first video segment is followed by the third videosegment in memory and that the third video segment is followed by thesecond video segment in memory.

7. The computer-implemented method of any of clauses 1-6, furthercomprising determining, based on the logical arrangement of videosegments, that the first portion of audio data includes audio samplescorresponding to the third video segment and that the second portion ofaudio data includes audio samples corresponding to both the second videosegment and the third video segment.

8. The computer-implemented method of any of clauses 1-7, whereinpreventing the first portion of audio data from being played back inconjunction with the first sequence of video segments comprisesreplacing the first portion of audio data with a third portion of audiodata corresponding to either the first video segment or the second videosegment 9.

10. The computer-implemented method of any of clauses 1-9, wherein themedia title comprises a branching narrative media title, and wherein thelogical arrangement of video segments corresponds to a plurality ofdifferent story arcs.

11. Some embodiments include a non-transitory computer-readable mediumstoring program instructions that, when executed by a processor, causethe process to perform the steps of identifying a first portion of audiodata that should not be played back in conjunction with a first sequenceof video segments associated with a media title, identifying a secondportion of audio data that also should not be played back in conjunctionwith the first sequence of video segments, determining that the firstportion of audio data has a greater playback duration than the secondportion of audio data, and preventing the first portion of audio datafrom being played back in conjunction with a first transition betweentwo segments included in the first sequence of video segments.

12. The non-transitory computer-readable medium of clause 11, whereinthe step of preventing the first portion of audio data from being playedback in conjunction with the first sequence of video segments comprisescanceling one or more playback operations involving a first audio frame,wherein the first audio frame includes the first portion of audio data,and the first audio frame overlaps a first physical boundary that isassociated with a first video segment included in the first sequence ofsegments and a second video segment included in the first sequence ofsegments.

13. The non-transitory computer-readable medium of any of clauses 11-12,wherein the first transition comprises a logical transition between thefirst video segment and a third video segment.

14. The non-transitory computer-readable medium of any of clauses 11-13,wherein a logical arrangement of video segments associated with themedia title indicates that the first sequence of video segments includesa first video segment followed by a second video segment and that asecond sequence of video segments includes the first video segmentfollowed by a third video segment.

15. The non-transitory computer-readable medium of any of clauses 11-14,further comprising the step of determining, based on the logicalarrangement of video segments, that the first portion of audio dataincludes audio samples corresponding to the third video segment and thatthe second portion of audio data includes audio samples corresponding toboth the second video segment and the third video segment.

16. The non-transitory computer-readable medium of any of clauses 11-15,wherein the step of preventing the first portion of audio data frombeing played back in conjunction with the first sequence of videosegments comprises replacing the first portion of audio data with athird portion of audio data corresponding to either the first videosegment or the second video segment 17.

18. The non-transitory computer-readable medium of any of clauses 11-17,wherein a logical arrangement of video segments associated with themedia title indicates that the first sequence of video segments includesa first video segment followed by a second video segment and that asecond sequence of video segments includes a third video segmentfollowed by the second video segment.

19. The non-transitory computer-readable medium of any of clauses 11-18,wherein a physical arrangement of video segments associated with themedia title indicates that the first video segment is followed by thethird video segment in memory and that the third video segment isfollowed by the second video segment in memory.

20. Some embodiments include a system, comprising a memory storing aplayback application, and a processor that, when executing the playbackapplication, is configured to perform the steps of identifying a firstportion of audio data that should not be played back in conjunction witha first sequence of video segments associated with a media title,identifying a second portion of audio data that also should not beplayed back in conjunction with the first sequence of video segments,determining that the first portion of audio data has a greater playbackduration than the second portion of audio data, and preventing the firstportion of audio data from being played back in conjunction with a firsttransition between two segments included in the first sequence of videosegments.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present embodimentsand protection

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a ““module” or“system.” Furthermore, aspects of the present disclosure may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine. The instructions, when executed via the processor ofthe computer or other programmable data processing apparatus, enable theimplementation of the functions/acts specified in the flowchart and/orblock diagram block or blocks. Such processors may be, withoutlimitation, general purpose processors, special-purpose processors,application-specific processors, or field-programmable gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow

What is claimed is:
 1. A computer-implemented method, comprising:identifying a first portion of audio data that does not correspond toany portions of video data included in a first sequence of videosegments associated with a media title; identifying a second portion ofaudio data that also does not correspond to any portions of video dataincluded in the first sequence of video segments; determining that thefirst portion of audio data has a greater playback duration than thesecond portion of audio data; and in response to determining that thefirst portion of audio data has greater playback duration, preventingthe first portion of audio data from being played back in conjunctionwith the first sequence of video segments.
 2. The computer-implementedmethod of claim 1, wherein preventing the first portion of audio datafrom being played back in conjunction with the first sequence of videosegments comprises canceling one or more playback operations involving afirst audio frame, wherein the first audio frame includes the firstportion of audio data, and the first audio frame overlaps a physicalboundary associated with a first video segment included in the firstsequence of segments and a second video segment included in the firstsequence of segments.
 3. The computer-implemented method of claim 1,further comprising performing a cross-fade operation between the secondportion of audio data and a third portion of audio data that should beplayed back in conjunction with the first sequence of video segments. 4.The computer-implemented method of claim 1, further comprisingperforming a transition operation between the second portion of audiodata and a third portion of audio data that should be played back inconjunction with the first sequence of video segments.
 5. Thecomputer-implemented method of claim 1, wherein a logical arrangement ofvideo segments associated with the media title indicates that the firstsequence of video segments includes a first video segment followed by asecond video segment and that a second sequence of video segmentsincludes the first video segment followed by a third video segment. 6.The computer-implemented method of claim 5, wherein a physicalarrangement of video segments associated with the media title indicatesthat the first video segment is followed by the third video segment inmemory and that the third video segment is followed by the second videosegment in memory.
 7. The computer-implemented method of claim 5,further comprising determining, based on the logical arrangement ofvideo segments, that the first portion of audio data includes audiosamples corresponding to the third video segment and that the secondportion of audio data includes audio samples corresponding to both thesecond video segment and the third video segment.
 8. Thecomputer-implemented method of claim 5, wherein preventing the firstportion of audio data from being played back in conjunction with thefirst sequence of video segments comprises replacing the first portionof audio data with a third portion of audio data corresponding to eitherthe first video segment or the second video segment.
 9. Thecomputer-implemented method of claim 5, wherein preventing the firstportion of audio data from being played back in conjunction with thefirst sequence of video segments comprises: moving an audio transitionpoint associated with both the first portion of audio data and thesecond portion of audio data relative to a video transition pointassociated with both the first video segment and the second videosegment; and causing a third portion of audio data corresponding to thethird video segment to be played back in conjunction with the secondvideo segment, wherein the third portion of audio data includes audiosamples corresponding to both the second video segment and the thirdvideo segment.
 10. The computer-implemented method of claim 5, whereinthe media title comprises a branching narrative media title, and whereinthe logical arrangement of video segments corresponds to a plurality ofdifferent story arcs.
 11. One or more non-transitory computer-readablemedia storing program instructions that, when executed by one or moreprocessors, cause the one or more processors to perform the steps of:identifying a first portion of audio data that does not correspond toany portions of video data included in a first sequence of videosegments associated with a media title; identifying a second portion ofaudio data that also does not correspond to any portions of video dataincluded in the first sequence of video segments; determining that thefirst portion of audio data has a greater playback duration than thesecond portion of audio data; and in response to determining that thefirst portion of audio data has a greater playback duration, preventingthe first portion of audio data from being played back in conjunctionwith a first transition between two segments included in the firstsequence of video segments.
 12. The one or more non-transitorycomputer-readable media of claim 11, wherein the step of preventing thefirst portion of audio data from being played back in conjunction withthe first sequence of video segments comprises canceling one or moreplayback operations involving a first audio frame, wherein the firstaudio frame includes the first portion of audio data, and the firstaudio frame overlaps a first physical boundary that is associated with afirst video segment included in the first sequence of segments and asecond video segment included in the first sequence of segments.
 13. Theone or more non-transitory computer-readable media of claim 12, whereinthe first transition comprises a logical transition between the firstvideo segment and a third video segment.
 14. The one or morenon-transitory computer-readable media of claim 11, wherein a logicalarrangement of video segments associated with the media title indicatesthat the first sequence of video segments includes a first video segmentfollowed by a second video segment and that a second sequence of videosegments includes the first video segment followed by a third videosegment.
 15. The one or more non-transitory computer-readable media ofclaim 14, further comprising the step of determining, based on thelogical arrangement of video segments, that the first portion of audiodata includes audio samples corresponding to the third video segment andthat the second portion of audio data includes audio samplescorresponding to both the second video segment and the third videosegment.
 16. The one or more non-transitory computer-readable media ofclaim 14, wherein the step of preventing the first portion of audio datafrom being played back in conjunction with the first sequence of videosegments comprises replacing the first portion of audio data with athird portion of audio data corresponding to either the first videosegment or the second video segment.
 17. The one or more non-transitorycomputer-readable media of claim 14, wherein the step of preventing thefirst portion of audio data from being played back in conjunction withthe first sequence of video segments comprises: moving an audiotransition point associated with both the first portion of audio dataand the second portion of audio data relative to a video transitionpoint associated with both the first video segment and the second videosegment; and causing a third portion of audio data corresponding to thethird video segment to be played back in conjunction with the secondvideo segment, wherein the third portion of audio data includes audiosamples corresponding to both the second video segment and the thirdvideo segment.
 18. The one or more non-transitory computer-readablemedia of claim 11, wherein a logical arrangement of video segmentsassociated with the media title indicates that the first sequence ofvideo segments includes a first video segment followed by a second videosegment and that a second sequence of video segments includes a thirdvideo segment followed by the second video segment.
 19. The one or morenon-transitory computer-readable media of claim 18, wherein a physicalarrangement of video segments associated with the media title indicatesthat the first video segment is followed by the third video segment inmemory and that the third video segment is followed by the second videosegment in memory.
 20. A system, comprising: one or more memoriesstoring a playback application; and one or more processors that, whenexecuting the playback application, are configured to perform the stepsof: identifying a first portion of audio data that does not correspondto any portions of video data included in a first sequence of videosegments associated with a media title, identifying a second portion ofaudio data that also does not correspond to any portions of video dataincluded in the first sequence of video segments, determining that thefirst portion of audio data has a greater playback duration than thesecond portion of audio data, and in response to determining that thefirst portion of audio data has a greater playback duration, preventingthe first portion of audio data from being played back in conjunctionwith a first transition between two segments included in the firstsequence of video segments.